Skip to content

DeflateBuffer: handle empty chunks in feed_data without raising#13015

Open
HrachShah wants to merge 2 commits into
aio-libs:masterfrom
HrachShah:fix/deflate-buffer-empty-chunk-v2
Open

DeflateBuffer: handle empty chunks in feed_data without raising#13015
HrachShah wants to merge 2 commits into
aio-libs:masterfrom
HrachShah:fix/deflate-buffer-empty-chunk-v2

Conversation

@HrachShah

@HrachShah HrachShah commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown

What do these changes do?

Fix :class:aiohttp.http_parser.DeflateBuffer (and the
chunked-transfer decoder that drives it) raising
IndexError when a paused decoder is resumed with an empty
chunk.

The chunked-transfer decoder calls :meth:feed_data with
b"" to resume a paused decompressor. The CM-byte sniff at
the top of :meth:feed_data read chunk[0] on the empty
chunk and raised :class:IndexError, which surfaced to the
HTTP parser as an unhandled exception during response decoding.

Skip the CM-byte sniff when the chunk is empty. The downstream
decompressor already handles empty input correctly, so no
behaviour change for non-empty chunks.

Are there changes in behavior for the user?

Yes, but only in the failure mode: the parser no longer crashes
when an empty chunk is fed in to resume a paused decoder. For
all non-empty input the behaviour is unchanged.

Is it a substantial burden for the maintainers to support this?

No. Two-line change guarded by an if not chunk check, with
the new test pinning the behaviour.

Related issue number

Fixes #12994

Checklist

  • I think the code is well written
  • Unit tests for the changes exist
  • Documentation reflects the changes (N/A — internal http_parser fix, no public API change)
  • If you provide code modification, please add yourself to CONTRIBUTORS.txt
  • Add a new news fragment into the CHANGES/ folder
    • name: 12994.bugfix.rst
    • category: bugfix

Drafted with Zo Bot; reviewed by @HrachShah.

DeflateBuffer: handle empty chunks in feed_data without raising
650925c 
The chunked-transfer decoder resumes a paused res
@psf-chronographer psf-chronographer Bot added the bot:chronographer:provided There is a change note present in this PR label Jun 30, 2026
@codecov

codecov Bot commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown

❌ 3 Tests Failed:

Tests completed Failed Passed Skipped
4836 3 4833 45
View the top 3 failed test(s) by shortest run time 
tests.test_http_parser.TestDeflateBuffer::test_feed_data_empty
Stack Traces | 0.008s run time 
self = <test_http_parser.TestDeflateBuffer object at 0x10c275590>
protocol = <NonCallableMagicMock spec_set='BaseProtocol' id='4486554064'>

    #x1B[0m#x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mtest_feed_data_empty#x1B[39;49;00m(#x1B[96mself#x1B[39;49;00m, protocol: BaseProtocol) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
    #x1B[90m    #x1B[39;49;00m#x1B[33m"""feed_data(b"") must not raise even on a fresh deflate stream.#x1B[39;49;00m
    #x1B[33m#x1B[39;49;00m
    #x1B[33m    The chunked-transfer decoder calls feed_data(b"") to give a paused#x1B[39;49;00m
    #x1B[33m    decoder another chance to make progress. The CM-byte sniff in#x1B[39;49;00m
    #x1B[33m    feed_data would previously read chunk[0] on an empty chunk and raise#x1B[39;49;00m
    #x1B[33m    IndexError. Regression test for #12994.#x1B[39;49;00m
    #x1B[33m    """#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        buf = aiohttp.StreamReader(#x1B[90m#x1B[39;49;00m
            protocol, DEFAULT_CHUNK_SIZE, loop=asyncio.get_running_loop()#x1B[90m#x1B[39;49;00m
        )#x1B[90m#x1B[39;49;00m
        dbuf = DeflateBuffer(buf, #x1B[33m"#x1B[39;49;00m#x1B[33mdeflate#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        #x1B[90m# Should be a no-op, returning False (no more data).#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        #x1B[94massert#x1B[39;49;00m dbuf.feed_data(#x1B[33mb#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m) #x1B[95mis#x1B[39;49;00m #x1B[94mFalse#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        #x1B[90m# No bytes pushed to the downstream stream.#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        #x1B[94massert#x1B[39;49;00m #x1B[96mlist#x1B[39;49;00m(buf._buffer) == []#x1B[90m#x1B[39;49;00m
        #x1B[90m# Decoder was not switched to suppress_deflate_header by the empty#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        #x1B[90m# chunk.#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
>       #x1B[94massert#x1B[39;49;00m dbuf._started_decoding #x1B[95mis#x1B[39;49;00m #x1B[94mFalse#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE       assert True is False#x1B[0m
#x1B[1m#x1B[31mE        +  where True = <aiohttp.http_parser.DeflateBuffer object at 0x10b6b6ad0>._started_decoding#x1B[0m

buf        = <StreamReader>
dbuf       = <aiohttp.http_parser.DeflateBuffer object at 0x10b6b6ad0>
protocol   = <NonCallableMagicMock spec_set='BaseProtocol' id='4486554064'>
self       = <test_http_parser.TestDeflateBuffer object at 0x10c275590>

#x1B[1m#x1B[31mtests/test_http_parser.py#x1B[0m:3004: AssertionError
tests.test_http_parser::test_te_header_non_ascii[py-parser]
Stack Traces | 0.013s run time 
parser = <aiohttp.http_parser.HttpRequestParser object at 0x10f1b3e50>

    #x1B[0m#x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mtest_te_header_non_ascii#x1B[39;49;00m(parser: HttpRequestParser) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
        #x1B[90m# K = Kelvin sign, not valid ascii.#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        text = #x1B[33m"#x1B[39;49;00m#x1B[33mGET /test HTTP/1.1#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33mHost: a#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33mTransfer-Encoding: chunKed#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
>       #x1B[94mwith#x1B[39;49;00m pytest.raises(http_exceptions.BadHttpMessage):#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE       Failed: DID NOT RAISE BadHttpMessage#x1B[0m

parser     = <aiohttp.http_parser.HttpRequestParser object at 0x10f1b3e50>
text       = 'GET /test HTTP/1.1\r\nHost: a\r\nTransfer-Encoding: chunKed\r\n\r\n'

#x1B[1m#x1B[31mtests/test_http_parser.py#x1B[0m:862: Failed
tests.test_http_parser::test_te_header_non_ascii[c-parser]
Stack Traces | 0.015s run time 
parser = <aiohttp._http_parser.HttpRequestParser object at 0x10e1325a0>

    #x1B[0m#x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mtest_te_header_non_ascii#x1B[39;49;00m(parser: HttpRequestParser) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
        #x1B[90m# K = Kelvin sign, not valid ascii.#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        text = #x1B[33m"#x1B[39;49;00m#x1B[33mGET /test HTTP/1.1#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33mHost: a#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33mTransfer-Encoding: chunKed#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33m\r#x1B[39;49;00m#x1B[33m\n#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
>       #x1B[94mwith#x1B[39;49;00m pytest.raises(http_exceptions.BadHttpMessage):#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE       Failed: DID NOT RAISE BadHttpMessage#x1B[0m

parser     = <aiohttp._http_parser.HttpRequestParser object at 0x10e1325a0>
text       = 'GET /test HTTP/1.1\r\nHost: a\r\nTransfer-Encoding: chunKed\r\n\r\n'

#x1B[1m#x1B[31mtests/test_http_parser.py#x1B[0m:862: Failed

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@codspeed-hq

codspeed-hq Bot commented Jun 30, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 83 untouched benchmarks
⏩ 83 skipped benchmarks1

Comparing HrachShah:fix/deflate-buffer-empty-chunk-v2 (42c32d3) with master (6b018d5)2

Open in CodSpeed

Footnotes

  1. 83 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on master (664d695) during the generation of this report, so 6b018d5 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@HrachShah HrachShah marked this pull request as ready for review June 30, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@webknjaz webknjaz Awaiting requested review from webknjaz webknjaz is a code owner
@asvetlov asvetlov Awaiting requested review from asvetlov asvetlov is a code owner

Assignees

No one assigned

Labels

bot:chronographer:provided There is a change note present in this PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

DeflateBuffer.feed_data raises IndexError on empty chunks

1 participant

@HrachShah